United  States  General  Accounting  Office 


GAD 


Report  to  the  Chairman,  Subcommittee 

on  Military  Research  and  Development, 
Committee  on  National  Security,  House 
of  Representatives 


October  1998 


DEFENSE 

ACQUISITION 

Advanced  Concept 
Technology 
Demonstration 
Program  Can  Be 
Improved 


Reproduced  From 
Best  Available  Copy 


Us 


23SSTP22U 


no 


OJ 


GAO/NSIAD-99-4 


GAD 


United  States 

General  Accounting  Office 

Washington,  D.C.  20548 


National  Security  and 
International  Affairs  Division 

B-278429 

October  15, 1998 

The  Honorable  Curt  Weldon 
Chairman,  Subcommittee  on  Military 
Research  and  Development 
Committee  on  National  Security 
House  of  Representatives 

Dear  Mr.  Chairman: 

The  Advanced  Concept  Technology  Demonstration  (actd)  program  was 
initiated  by  the  Department  of  Defense  (dod)  to  determine  the  extent  to 
which  a  given  mature  technology  will  improve  military  capabilities  before 
entering  the  normal  acquisition  process  and,  by  using  mature  technology, 
reduce  the  length  of  time  to  develop  and  acquire  weapon  systems. 
According  to  the  actd  program  documentation,  the  actd  program  was 
established  in  1994  in  response  to  recommendations  of  the  1986  Packard 
Commission1  and  a  1991  Defense  Science  Board  study.2  The  Packard 
Commission  concluded  that  major  improvements  could  be  made  in 
defense  acquisition  by  emulating  the  practices  of  successful  commercial 
companies.  Accordingly,  the  Commission  recommended,  among  other 
things,  building  and  testing  prototypes  to  assess  military  utility  and 
provide  a  basis  for  realistic  cost  estimates  before  committing  to 
acquisition.  The  Defense  Science  Board’s  study  discussed  the  need  for 
early  dialogue  between  the  potential  system’s  user  and  producer  for  a 
proper  analysis  of  cost,  risk,  and  operational  capability. 

actds  have  been  the  subject  of  congressional  interest  since  the  program’s 
inception.  Congressional  committees  have  expressed  concerns  about  the 
validity  of  several  technologies  selected  for  the  actd  program  and  the 
number  of  assets  procured  for  the  projects.  At  your  request,  we  assessed 
the  current  actd  program.  Specifically,  we  determined  whether  (1)  the 
selection  process  includes  criteria  that  are  adequate  to  ensure  that  only 
mature  technologies  are  selected  for  actd  prototypes  and  (2)  guidance  on 
transitioning  to  the  normal  acquisition  process  ensures  that  a  prototype 
appropriately  completes  product  and  concept  development  and  testing 
before  entering  production.  Finally,  we  assessed  dod’s  current  practice  of 


lrThe  Packard  Commission  was  created  by  former  President  Reagan  in  1985  to  review  the  defense 
acquisition  system  to  determine  how  military  acquisitions  could  be  made  quicker  and  at  lower  costs. 

2Report  of  the  Defense  Science  Board,  1991  Summer  Study  on  Weapon  Development  and  Production 
Technology,  November  1991. 
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procuring  more  actd  prototypes  than  needed  to  assess  the  military  utility 
of  a  mature  technology. 


Background 


Through  fiscal  year  1998,  about  $172  million  has  been  allocated  to  the  actd 
program  and  48  projects  have  been  approved,  dod’s  budget  request  for 
fiscal  year  1999  for  the  actd  program  is  $116.4  million.  An  additional  10  to 
15  projects  are  expected  to  be  funded  in  fiscal  year  1999. 

Under  the  current  actd  program,  dod  builds  prototypes  to  assess  the 
military  utility  of  mature  technologies,  which  are  used  to  reduce  or  avoid 
the  time  and  effort  usually  devoted  to  technology  development. 
Demonstrations  that  assess  a  prototype’s  military  utility  are  structured  to 
be  completed  within  2  to  4  years  and  require  the  participation  of  field 
users  (war  fighters). 

actd  projects  are  not  acquisition  programs.  The  actd  program  seeks  to 
provide  the  war  fighter  with  the  opportunity  to  assess  a  prototype’s 
capability  in  realistic  operational  scenarios.  From  this  demonstration,  the 
war  fighter  can  refine  operational  requirements,  develop  an  initial  concept 
of  operation,  and  make  a  determination  of  the  military  utility  of  the 
technology  before  dod  decides  whether  the  technology  should  enter  into 
the  normal  acquisition  process. 

Not  all  projects  will  be  selected  for  transition  into  the  normal  acquisition 
process.  The  user  can  conclude  that  the  technology  (1)  does  not  have 
sufficient  military  utility  and  that  acquisition  is  not  warranted  or  (2)  has 
sufficient  utility  but  that  additional  procurement  is  not  necessary.  Of  the 
11  actd  projects  completed  as  of  August  1998,  2  were  found  to  have 
insufficient  utility  to  proceed  further,  8  were  found  to  have  military  utility 
but  no  further  procurement  was  found  to  be  needed  at  the  time,3  and  1 
was  found  to  have  utility  and  has  transitioned  to  the  normal  acquisition 
process. 

actd  funding  is  to  be  used  to  procure  enough  prototypes  to  conduct  the 
basic  demonstration  of  military  utility.  At  the  conclusion  of  the  basic 
demonstration,  actd  projects  are  expected  to  provide  a  residual 
operational  capability  for  the  war  fighter.  Under  the  current  practice,  actd 
funding  is  also  to  be  available  to  support  continued  use  of  actd  prototypes 
that  have  military  utility  for  a  2-year,  post-demonstration  period.  The 


3Three  of  the  projects  in  this  category  are  primarily  software,  for  which  production  would  not  be 
appropriate. 
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2  years  of  funding  is  to  support  continued  use  by  an  operational  unit  and 
provide  the  time  needed  to  separately  budget  for  the  acquisition  of 
additional  systems.  Further,  if  the  actd  prototypes— such  as  missiles— will 
be  consumed  during  the  basic  demonstration,  additional  prototypes  are  to 
be  procured. 

As  stated  in  the  actd  guidance,  a  key  to  successfully  exploiting  the  results 
of  the  demonstration  is  to  enter  the  appropriate  phase  of  acquisition 
without  loss  of  momentum,  actds  are  intended  to  shorten  the  acquisition 
cycle  by  reducing  or  eliminating  technology  development  and  maturation 
activities  during  the  normal  acquisition  process.  Further,  dod  can 
concentrate  more  on  technology  integration  and  demonstration  activities. 
Time  and  effort  usually  devoted  to  technology  development  can  be 
significantly  reduced  or  avoided  and  the  subsequent  acquisition  process 
reduced  accordingly,  if  the  project  is  deemed  to  have  sufficient  military 
utility. 

actd  candidates  are  nominated  from  a  variety  of  sources  within  the 
defense  community,  including  the  Commanders  in  Chief,  the  Joint  Chiefs 
of  Staff,  the  Office  of  the  Secretary  of  Defense  agencies,  the  services,  and 
the  research  and  development  laboratories.  The  candidates  are  then 
reviewed  and  assessed  by  staff  from  the  Office  of  the  Deputy  Under 
Secretary  of  Defense  (Advanced  Technology).  After  this  initial  screening, 
the  remaining  candidates  are  further  assessed  by  a  panel  of  technology 
experts.  The  best  candidates  are  then  submitted  to  the  Joint  Requirements 
Oversight  Council,  which  assesses  their  priority.  The  final  determination 
of  the  candidates  to  be  funded  is  made  within  the  Office  of  the  Deputy 
Under  Secretary  of  Defense  (Advanced  Technology),  with  final  approval 
by  the  Under  Secretary  of  Defense  (Acquisition  and  Technology). 


Results  in  Brief 


Through  the  determination  of  military  value  of  mature  technologies  and 
their  use  in  the  acquisition  process,  actds  have  the  potential  to  reduce  the 
time  to  develop  and  acquire  weapon  systems.  However,  several  aspects  of 
the  actd  program  can  be  improved. 

dod’s  process  for  selecting  actd  candidates  does  not  include  adequate 
criteria  for  assessing  the  maturity  of  the  proposed  technology  and  has 
resulted  in  the  approval  of  actd  projects  that  included  immature 
technology,  dod  has  improved  its  guidance  on  the  maturity  of  the 
technologies  to  be  used  in  actd  projects  but  the  revised  guidance 
describes  several  types  of  exceptions  under  which  immature  technologies 
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may  be  used.  Where  dod  approves  immature  technologies  as  actd  program 
candidates  and  time  is  spent  conducting  developmental  activities,  the  goal 
of  reduced  acquisition  cycle  time  will  not  be  realized. 

Further,  guidance  on  entering  technologies  into  the  normal  acquisition 
process  is  not  sufficient  to  ensure  that  a  prototype  completes  product  and 
concept  development  and  testing  before  entering  production.  According  to 
the  actd  guidance,  if  the  prototype  is  found  to  have  military  utility,  actd 
can  directly  enter  production.  The  guidance  does  not  mention  the 
circumstances  when  transition  to  development  may  be  appropriate  or  the 
kinds  of  developmental  activities  that  may  be  appropriate.  While 
commercial  items  that  do  not  require  any  further  development  could 
proceed  directly  to  production,  many  actds  may  still  need  to  enter  the 
engineering  and  manufacturing  development  phase  to  proceed  with 
product  and  concept  development  and  testing  before  production  begins. 
Through  the  actd  early  user  demonstration,  dod  is  expected  to  obtain 
more  detailed  knowledge  about  its  technologies  before  entering  into  the 
acquisition  process.  However,  in  the  one  case  in  which  an  actd  has 
proceeded  into  production,  dod  made  that  decision  before  completing 
product  and  concept  development  and  testing,  thereby  accepting 
programmatic  risks  that  could  offset  the  schedule  and  other  benefits 
gained  through  the  actd  process. 

dod’s  current  practice  of  procuring  prototypes  beyond  those  needed  for 
the  basic  actd  demonstration  and  before  completing  product  and  concept 
development  and  testing  is  unnecessarily  risky.  This  practice  risks  wasting 
resources  on  the  procurement  of  items  that  may  not  work  as  expected  or 
may  not  have  sufficient  military  utility — as  would  be  determined  in  the 
basic  demonstration — and  risks  a  premature  and  excessive  commitment  to 
production. 


Inadequate  Guidance 
for  Assessing  Maturity 
of  Technology 


By  limiting  consideration  to  prototypes  that  feature  mature  technology, 
the  actd  program  avoids  the  time  and  risks  associated  with  technology 
development,  concentrating  instead  on  technology  integration  and 
demonstration  activities.  The  information  gained  through  the 
demonstration  of  the  mature  technology  could  provide  a  good  jump  start 
to  the  normal  acquisition  process,  if  the  demonstration  shows  that  the 
technology  has  sufficient  military  value.  Time  and  effort  usually  devoted  to 
technology  development  could  be  reduced  or  avoided  and  the  acquisition 
process  shortened  accordingly. 
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Program  officials  stated  that  they  have  a  mechanism  in  place  to  ensure 
that  only  those  projects  using  mature  technology  are  allowed  to  become 
actds.  These  officials  explained  that  an  actd  candidate’s  technology  is 
assessed  by  high-ranking  representatives  from  the  services  and  the  dod 
science  and  technology  community  before  candidates  are  selected. 
Program  personnel  stated  that  determining  technology  maturity  is 
important  before  a  candidate  is  selected  because  actd  program  funding  is 
not  intended  to  be  used  for  technology  development.  According  to 
program  guidance,  the  actd  funding  is  to  be  used  for  (1)  costs  incurred 
when  existing  technology  programs  are  reoriented  to  support  actd, 

(2)  costs  to  procure  additional  assets  for  the  basic  actd  demonstration, 
and  (3)  costs  for  technical  support  for  2  years  of  field  operations  following 
the  basic  actd  demonstration.  We  were  told  that  no  actd  money  was  to  be 
used  for  technology  development  activities. 

However,  the  project  selection  process  does  not  ensure  that  only  mature 
technologies  enter  the  actd  program.  We  found  examples  where  immature 
technologies  were  selected  and  technology  development  was  taking  place 
after  the  approval  and  start  of  the  actd  program.  The  current  operations 
manager  of  the  Combat  Identification  project,  which  began  in  fiscal  year 
1996,  told  us  that  one  of  his  major  concerns  has  been  that  some  of  the  actd 
funding  was  being  used  for  technology  development,  and  not  exclusively 
used  for  designing  and  implementing  the  assessment.  However,  during  the 
actd  project,  technical  or  laboratory  testing  was  still  necessary  to  evaluate 
the  acceptability  of  many  of  the  12  technologies  included  in  the  initial 
project.  Eventually,  6  of  the  12  technologies  had  to  be  terminated. 
According  to  the  demonstration  manager,  2  of  the  6  technologies  were 
terminated  because  they  were  immature.  According  to  the  manager,  that  is 
one  of  the  reasons  the  project  is  currently  behind  schedule. 

Another  example  of  the  inclusion  of  immature  technology  occurred  in  the 
Outrider  Unmanned  Aerial  Vehicle  project.  According  to  the  management 
plan  for  the  project,  one  of  the  individual  technologies  to  be  incorporated 
into  the  vehicle  was  a  heavy  fuel  engine.  According  to  a  program  official,  it 
was  later  deemed  that  this  individual  technology  was  too  immature  and  an 
alternate  technology  had  to  be  used.  However,  trying  to  use  this  immature 
technology  has  already  caused  schedule  slippage  and  cost  overruns  in  the 
actd  project. 
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Insufficient  Guidance 
for  Transitioning 
ACTDs  to  the  Normal 
Acquisition  Process 


To  complete  the  basic  demonstration  within  the  prescribed  2  to  4  year 
period,  actds  typically  use  early  prototypes.  If  the  demonstrated 
technology  is  deemed  to  have  sufficient  military  utility,  many  actd  projects 
will  still  need  to  enter  the  normal  acquisition  process  to  complete  product 
and  concept  development  and  testing  to  determine,  for  example,  whether 
the  system  is  producible  and  can  meet  the  user’s  suitability  needs.4  These 
attributes  of  a  system  go  beyond  the  actd’s  demonstration  of  military 
utility  to  address  whether  the  item  can  meet  the  full  military  requirement. 
Commercial  items  that  do  not  require  any  further  development  could 
proceed  directly  to  production.  However,  other  non-software  related  actds 
should  enter  the  engineering  and  manufacturing  development  phase  to 
proceed  with  product  and  concept  development  and  testing. 


According  to  actd  guidance,  if  further  significant  development  is  needed,  a 
system  might  enter  the  development  portion  of  the  engineering  and 
manufacturing  development  phase.  However,  the  guidance  states  that,  if 
the  capability  is  adequate,  the  actd  can  directly  enter  production.  The 
guidance  does  not  specifically  define  what  is  considered  an  “adequate 
capability”  to  allow  an  actd  system  to  enter  low-rate  production. 


In  1994,  we  reported  on  numerous  instances  of  weapon  systems  that  began 
production  prematurely  and  later  experienced  significant  operational 
effectiveness  or  suitability  problems.5  In  our  best  practices  report,  we 
reported  that  typically  dod  programs  allowed  much  more  technology 
development  to  continue  into  the  product  development  phase  than  is  the 
case  in  commercial  practices.6  Turbulence  in  program  outcomes — in  the 
form  of  production  problems  and  associated  cost  and  schedule 
increases — was  the  predictable  consequence  of  dod’s  actions.  In  contrast, 
commercial  firms  gained  more  knowledge  about  a  product’s  technology, 
performance,  and  producibility  much  earlier  in  the  product  development 
process.  Commercial  firms  consider  not  having  this  type  of  knowledge 
early  in  the  acquisition  process  an  unacceptable  risk.  In  responding  to  that 
report,  the  Secretary  of  Defense  stated  that  dod  is  vigorously  pursuing  the 
adoption  of  such  business  practices.  Specifically,  he  stated  that  dod  has 
taken  steps  to  separate  technology  development  from  product 
development  through  the  use  of  actds.  The  actd  guidance  and  dod’s 
current  practice  do  not  appear  to  reflect  this  emphasis. 


4“Suitability”  involves  factors  such  as  maintainability,  reliability,  safety,  and  supportability. 

5Weapons  Acquisition:  Low-Rate  Initial  Production  Used  to  Buy  Weapon  Systems  Prematurely 
(GAO/NSIAD-95-18,  Nov.  21, 1994). 

^Best  Practices:  Successful  Application  to  Weapon  Acquisitions  Requires  Changes  in  DOD’s 
Environment  (GAO/NSIAD-98-56,  Feb.  24,  1998). 
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In  the  case  of  the  Predator  actd,  the  one  actd  that  has  proceeded  into 
production,  dod  decided  to  enter  the  technology  into  production  before 
proceeding  with  product  and  concept  development  and  testing,  thereby 
accepting  programmatic  risks  that  could  offset  the  schedule  and  other 
benefits  gained  through  the  actd  process.  In  the  early  operational 
assessment  of  the  Predator’s  actd  demonstration,  the  Director, 

Operational  Test  and  Evaluation,  did  not  make  a  determination  of  the 
system’s  potential  operational  effectiveness  or  suitability.  However,  the 
system  was  found  to  be  deficient  in  several  areas,  including  mission 
reliability,  documentation,  and  pilot  training.  The  assessment  also  noted 
that  the  actd  demonstration  was  not  designed  to  evaluate  several  other 
areas  such  as  system  survivability,  supportability,  target  location  accuracy, 
training,  and  staffing  requirements. 

The  basic  actd  demonstration  may  have  clarified  the  Predator’s  military 
utility  but  it  did  not  demonstrate  its  system  requirements  or  its  suitability. 
Thus,  instead  of  using  the  knowledge  acquired  during  the  demonstration 
to  complete  the  Predator’s  development  through  the  product  and  concept 
development  and  testing  stages  of  acquisition,  dod  allowed  it  to  directly 
enter  production. 


Procuring  ACTD 
Prototypes  Beyond 
Those  Needed  for 
Basic  Demonstration 
Is  Unnecessarily 
Risky 


DOD’s  practice  is  to  procure  sufficient  actd  prototypes  to  provide  a  2-year 
residual  capability.  When  it  determines  that  the  original  prototypes  will  be 
consumed  during  the  basic  demonstration,  additional  prototypes  are 
procured  for  potential  use  after  the  basic  actd  demonstration.  However, 
these  additional  assets — like  the  basic  demonstration  prototypes — have 
not  been  independently  tested  to  determine  their  effectiveness  and 
suitability.  Procuring  additional  actd  prototypes  before  product  and 
concept  development  and  testing  is  completed  risks  wasting  resources  on 
the  procurement  of  items  that  may  not  work  as  expected  or  may  not  have 
sufficient  military  utility.  Representatives  from  the  service  test  agencies 
did  not  support  this  practice  and  agreed  that  it  had  the  potential  for 
problems.  Without  a  meaningful  independent  assessment  of  a  product’s 
suitability,  effectiveness,  and  survivability,  users  cannot  be  assured  that  it 
will  operate  as  intended  and  is  supportable. 

Congress  has  expressed  concern  about  the  amount  of  equipment  being 
procured  beyond  what  is  needed  to  conduct  the  basic  actd  demonstration. 
Its  concern  is  that  dod  is  making  an  excessive  commitment  to  production 
before  militaiy  utility  is  demonstrated  and  before  appropriate  concepts  of 
operation  are  developed.  For  example,  dod  plans  to  procure  192  Enhanced 
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Fiber  Optic  Guided  missiles  at  an  estimated  cost  of  $27  million  and  144 
Line-of-Sight  Anti-Tank  missiles  at  an  estimated  cost  of  $28  million  beyond 
the  quantities  of  missiles  required  for  the  actd  demonstrations — 64  and  30 
missiles,  respectively.  The  production  of  these  additional  missiles  will 
follow  the  production  of  the  missiles  needed  for  the  basic  demonstration 
and  will  continue  on  a  regular  basis  throughout  the  2-year, 
post-demonstration  period.  If  the  prototypes  cure  deemed  to  have  sufficient 
military  utility,  the  service  involved  will  be  expected  to  fund  the 
production  of  additional  missiles  beyond  these  quantities.  By  establishing 
a  regular  pattern  of  procurement  in  this  way,  dod  risks  committing  to  a 
continuing  production  program  before  a  determination  is  made  about  the 
technology’s  military  utility  and  before  there  is  assurance  that  the  system 
will  meet  validated  requirements  and  be  supportable. 


Conclusions 


The  strength  of  the  actd  program  is  in  conducting  basic  demonstrations  of 
mature  technology  in  military  applications  before  entering  the  normal 
acquisition  process.  This  practice  could  significantly  reduce  or  eliminate 
the  time  and  effort  needed  for  technology  development  from  the 
acquisition  process.  For  this  to  occur,  it  is  essential  that  dod  use  only 
mature  technology  in  its  actds.  dod’s  criteria  for  selecting  technologies  for 
actd  candidates  should  be  clarified  to  ensure  the  selection  of  mature 
technology  with  few,  if  any,  exceptions. 

Further,  actds  may  not,  by  themselves,  result  in  an  effective  and  safe 
deployment  of  military  capability.  It  is  important  that  product  and  concept 
development  as  well  as  test  and  evaluation  processes  be  allowed  to 
proceed  before  the  service  commits  to  the  production  of  the  demonstrated 
technology.  If  an  actd  project  is  shown  to  have  military  value,  the  normal 
acquisition  processes  can  and  should  be  tailored — but  not 
bypassed — before  dod  begins  production.  Lastly,  emphasizing  the  need  to 
complete  concept  and  product  development  and  testing  before  procuring 
more  items  than  needed  for  the  basic  demonstration  would  reduce  the  risk 
of  prematurely  starting  production. 


Recommendations 


We  recommend  that  the  Secretary  of  Defense  clarify  the  actd  program 
guidance  to  (1)  ensure  the  use  of  mature  technology  with  few,  if  any, 
exceptions  and  (2)  describe  when  transition  to  the  development  phase  of 
the  acquisition  cycle  is  necessary  and  the  types  of  development  activity 
that  may  be  appropriate.  Further,  we  recommend  that  the  Secretary  of 
Defense  limit  the  number  of  prototypes  to  be  procured  to  the  quantities 


Page  8 


GAO/NS  IAD-99-4  Defense  Acquisition 


B-278429 


needed  for  early  user  demonstrations  of  mature  technology  until  the  item’s 
product  and  concept  development  and  testing  have  been  completed. 


Agency  Comments 
and  Our  Evaluation 


In  commenting  on  a  draft  of  this  report,  dod  partially  concurred  with  each 
of  our  recommendations,  dod  said  that  it  continues  to  refine  the  actd 
selection  and  implementation  process,  dod  agreed  that  the  actd  program 
should  focus  on  mature  technologies  and  stated  that  it  had  improved  its 
definitions  of  mature  technology,  dod’s  new  guidance  on  the  maturity  of 
technology  to  be  used  in  actds  states 

“. . .  new  technologies  proposed  for  incorporation  into  an  actd  should  not  be  in  the  6.1 
(basic  research)  or  6.2  (applied  research)  budget  categories.  Furthermore,  the  technologies 
must  have  been  successfully  demonstrated  at  the  subsystem  or  component  level  and  at  the 
required  performance  level  prior  to  the  start  of  the  actd.” 

While  this  guidance  is  improved  over  previous  versions,  the  new  guidance 
permits  the  selection  of  immature  technology — even  as  the  primary  or 
core  technology — provided  that  it  is  demonstrated  prior  to  the  actd 
demonstration.  Also,  some  recent  actd  projects  have  been  approved 
without  the  technologies  having  been  identified.  Moreover,  the  new 
guidance  goes  on  to  describe  several  types  of  exceptions  under  which 
immature  technologies  may  be  permitted  to  be  used  in  an  actd.  As  our 
report  states,  the  use  of  immature  technologies  has  delayed  programs  and 
we  continue  to  believe  dod  needs  to  focus  the  actd  program  on  the  use  of 
mature  technology  with  few,  if  any  exceptions. 

dod  also  agreed  that  some  but  not  all  actds  may  require  additional  product 
and  concept  development  before  proceeding  into  production,  dod  states 
that  a  mandatory  engineering  and  manufacturing  development  phase 
would  not  be  appropriate  for  all  actd  projects.  We  agree,  however,  the 
existing  actd  guidance  focuses  on  the  transition  directly  to  production  and 
provides  too  little  guidance  concerning  a  possible  transition  to 
development.  As  stated  in  our  recommendation,  the  guidance  should 
specify  when  a  transition  to  development  may  be  appropriate  and  the 
kinds  of  developmental  activities  that  may  be  appropriate. 

Finally,  dod  agreed  that  the  number  of  actd  prototypes  to  be  procured 
should  be  limited  until  the  Under  Secretary  can  confirm  that  sufficient 
testing  has  been  satisfactorily  completed  to  support  any  additional 
procurement.  We  agree  with  dod  that  test  results  should  form  the  basis  for 
starting  limited  procurement.  However,  dod’s  equating  a  determination  of 
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military  utility  (based  on  an  actd  demonstration)  with  a  determination  of  a 
system’s  readiness  to  begin  production  is  inappropriate  because 
production  decisions  require  more  testing  data.  We  have  long  held  the 
view  and  have  consistently  recommended  that  dod  use  extreme  caution  to 
avoid  premature  commitments  to  production. 


Scope  and 
Methodology 


To  determine  the  adequacy  of  the  actd  program’s  selection  criteria  in 
assessing  technology  maturity  and  guidance  for  transitioning  to  the  normal 
acquisition  process,  we  reviewed  existing  program  guidance,  published 
reports,  the  Office  of  the  Inspector  General’s  April  1997  actd  report,  and 
the  recommendations  of  the  1986  Packard  commission  and  the  1996 
Defense  Science  Board.  We  discussed  selection  criteria,  transitioning  to 
the  acquisition  process,  and  all  34  of  the  individual  actd  programs 
approved  through  fiscal  year  1997  with  representatives  from  the  Office  of 
the  Deputy  Under  Secretary  of  Defense  (Advanced  Technology), 
Washington,  D.C.;  the  Army’s  Deputy  Chief  of  Staff  for  Operations  and 
Plans,  Office  of  Science  and  Technology  Programs,  Washington,  D.C.;  the 
Air  Force’s  Director  for  Operational  Requirements,  Rosslyn,  Virginia;  the 
Navy’s  Requirements  and  Acquisition  Support  Branch,  Washington,  D.C.; 
the  Marine  Corps’  Combat  Development  Command  Office  of  Science  and 
Innovation,  Quantico,  Virginia;  the  Joint  Staffs  Acquisition  and 
Technology  Division  and  Requirements  Assessment  Integration  Division, 
Washington,  D.C.;  and  the  Office  of  the  Commander  in  Chief,  U.S.  Atlantic 
Command,  Norfolk,  Virginia. 

We  discussed  the  issue  of  procuring  additional  residual  assets  for  early 
deployment  with  representatives  from  dod’s  Office  of  the  Director, 
Operational  Test  and  Evaluation,  Washington,  D.C.;  the  Army’s  Test  and 
Evaluation  Management  Agency,  Washington,  D.C.;  the  Army’s  Operational 
Test  and  Evaluation  Command,  Alexandria,  Virginia;  the  Marine  Corps’ 
Operational  Test  and  Evaluation  Activity,  Quantico,  Virginia;  the  Air 
Force’s  Test  and  Evaluation  Directorate,  Washington,  D.C.;  and  the  Navy’s 
Commander,  Operational  Test  and  Evaluation  Force,  Norfolk,  Virginia. 

We  conducted  our  review  from  September  1997  to  July  1998  in  accordance 
with  generally  accepted  government  auditing  standards. 


As  agreed  with  your  office,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  of  this  report  until 
30  days  from  its  issue  date.  At  that  time,  we  will  send  copies  to  other 
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interested  congressional  committees;  the  Secretaries  of  Defense,  the 
Army,  the  Air  Force,  and  the  Navy;  the  Commandant  of  the  Marine  Corps; 
the  Director,  Office  of  Management  and  Budget;  and  other  interested 
parties.  We  will  also  make  copies  available  to  others  upon  request. 

Please  contact  me  at  (202)  512-4841,  if  you  or  your  staff  have  any  questions 
concerning  this  report.  The  major  contributors  to  this  report  were 
Bill  Graveline,  Laura  Durland,  and  John  Randall. 

Sincerely  yours, 


Louis  J.  Rodrigues 

Director,  Defense  Acquisitions  Issues 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


ACQUISmON  AND 
TECHNOLOGY 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 


3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3000 


Mr.  Louis  J.  Rodrigues 

Director,  Defense  Acquisitions  Issues 

National  Security  and  International  Affairs  Division 

U.S.  General  Accounting  Office 

Washington,  DC  20548 


AUG  3  I  1998 


Dear  Mr.  Rodrigues: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  draft  report,  ‘DEFENSE  ACQUISITION:  Advanced  Concept  Technology  Demonstration 
Program  Can  Be  Improved,’  dated  July  28,  1998  (GAO  Code  707299/OSD  Case  1659).  The 
attached  document  contains  both  technical  corrections  and  a  detailed  substantive  response  to  the 
draft  report. 

The  Department  of  Defense  partially  concurs  with  the  draft  report.  While  recognizing  the 
GAO’s  concerns  regarding  technological  maturity,  ACTD  transition  to  acquisition,  and  prototype 
procurement,  the  Department  believes  it  has  made  great  progress  in  these  areas.  In  some  cases, 
these  improvements  post-date  the  events  analyzed  by  the  GAO.  The  Department  continues  to 
refine  the  ACTD  selection  and  implementation  processes.  Detailed  DoD  comments  are  provided 
in  the  enclosure. 

The  Department  appreciates  the  opportunity  to  comment  on  the  draft  report. 

/Joseph  7  Eash,  HI 
Depubrunder  Secretary  of  Defense 
(Advanced  Technology) 
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See  comment  1. 


GAO  Draft  Report  dated  July  28, 1998.  GAO  Code  707299,  OSD  Code  1659 

‘DEFENSE  ACQUISITION:  Advanced  Concept  Technology  Demonstration  Program 

Can  Be  Improved’ 


Department  of  Defense  Comments 

Reference  the  GAO  finding  that :  “The  current  process  for  selecting  ACTD 
candidates  lacks  adequate  criteria  for  assessing  the  maturity  of  a  proposed  ACTD’s 
technology.”  (p.5) 

DoD  Response:  DoD  concurs  with  GAO’s  assertion  that  the  Department  lacked 
adequate  criteria  for  assessing  the  maturity  of  technology  of  ACTDs  initiated  in  FY95 
and  FY96.  However,  in  response  to  the  DoD  Inspector  General’s  report  dated 
April  7, 1997,  several  changes  were  implemented  in  the  selection  process.  In  1997,  a 
working  group  comprised  of  representatives  of  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  (OUSD(A&T)),  developers,  users,  testers,  and 
the  office  of  the  IG  was  convened  to  develop  criteria  for  assessing  technology  maturity. 
The  resulting  criteria  were  subsequently  used  to  re-examine  the  FY97  ACTDs  and  to 
evaluate  the  candidates  for  approval  in  FY98  and  FY99.  These  criteria  were  published  in 
the  Defense  Acquisition  Deskbook  in  April  1998  and  restated  in  the  1998  ACTD  Master 
Plan  and  on  the  ACTD  website.  The  ACTD  candidate  review  process,  including  a  final 
review  (termed  the  ‘Final  Scrub’),  is  conducted  just  prior  to  the  start  of  the  new  fiscal 
year.  This  final  scrub  of  each  candidate  is  focused  on  the  ACTD  selection  criteria,  as 
well  as  the  transition  strategy  and  proposed  ACTD  management  structure. 

The  Department’s  new  ACTD  criteria  states  that  for  a  technology  to  be  deemed 
mature,  it  must  be  demonstrated  at  the  required  performance  level  and  in  the  expected 
operating  environment.  All  ACTDs  started  since  the  establishment  of  the  new  criteria 
have  been  compliant,  and  subsequently  free  of  criticism  on  this  issue. 

Less  mature  technology  can  be  used  in  ACTDs,  only  in  situations  where  it  will 
have  no  impact  on  the  overall  demonstration  project.  There  are  two  examples  of  such 
non-critical  roles  for  immature  technologies.  The  first  is  the  case  involving  a 
“piggyback”  technology  (or  advanced  system)  that  is  not  an  integral  part  of  the  ACTD 
capability  being  evaluated,  will  not  be  part  of  the  utility  assessment,  and  will  not  be  left 
as  part  of  the  residual  capability.  These  advanced  systems  participate  in  ACTD 
exercises  and  demonstrations  on  a  non-interference  basis,  taking  advantage  of  low  cost 
data  collection  opportunities.  While  providing  important  insights,  they  are  not  critical  to 
the  overall  success  of  the  ACTD,  and  their  maturity  or  lack  of  maturity  is  not  a  factor.  If 
they  fail  to  perform,  they  are  removed  from  the  program  or  the  demonstration.  These 
“piggy-backed”  systems  could  be  prohibited  from  participating  in  ACTDs;  however,  their 
participation  has  no  negative  impact  on  the  ACTDs  and  their  exclusion  would  result  in 
the  loss  of  valuable  data  collection  opportunities. 
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See  comment  2. 


See  comment  2. 


See  comment  3. 


The  two  technologies  in  the  Combat  Identification  ACTD,  identified  on  page 
eight  of  the  draft  report  as  being  terminated  because  they  were  immature,  were  non- 
critical  advanced  systems.  They  were  not  an  integral  part  of  the  capability  being 
evaluated.  The  draft  GAO  report  states  that  .  .immature  technologies  were  selected  and 
technology  development  was  clearly  taking  place  after  the  approval  and  start  of  the 
ACTD  program.”  In  addition  the  draft  report  states:  .during  the  ACTD  project, 
technical  or  laboratory  testing  was  still  necessary  to  evaluate  the  acceptability  of  many  of 
the  12  technologies  included  in  the  initial  project.” 

Mature  technologies  such  as  the  Battlefield  Combat  Identification  System  (BCIS), 
the  Situational  Awareness  Data  Link  (SADL),  and  the  Single  Channel  Ground  and 
Airborne  Radio  System,  System  Improvement  Program  Plus  (SINCGARS  SIP(+))  served 
as  the  core  technologies  for  this  ACTD.  The  advanced  concepts  that  were  included  were 
present  on  a  noninterference  basis.  The  two  technologies  cited  by  the  GAO  were 
terminated  with  no  impact  to  the  core  systems  and  to  the  primary  objectives  of  this 
ACTD. 

Technical  and  laboratory  testing  did  occur  during  the  CID  ACTD.  Technical 
testing  is  done  with  even  the  most  mature  components  and  subsystems  to  ensure  they  are 
properly  integrated  and  are  operating  correctly  before  deployment  to  an  exercise.  It  is 
also  necessary  to  fully  characterize  and  document  subsystem  level  performance  in  order 
to  conduct  a  valid  assessment  of  the  integrated  system. 

Three  of  the  baseline  CID  technologies  are  scheduled  for  procurement  —including 
BCIS  and  SADL.  Two  technologies,  Mark  XII/GPS  and  Laser/RF  were  dropped  because 
they  did  not  meet  the  key  performance  parameters  for  CID;  even  though  these  are  mature 
technologies  which  have  been  deployed  on  operational  systems.  In  addition,  two 
technology  programs  were  recommended  for  termination  by  the  warfighters  because  they 
did  not  make  operational  sense,  not  because  the  technology  was  immature.  And  finally, 
two  advanced  systems  were  dropped  from  the  ACTD  and  it  was  recommended  that  they 
go  back  to  the  technology  base. 

The  Department’s  ACTD  technological  maturity  criteria  also  permits  less  mature 
technologies  to  be  included  if  those  systems  or  components  are  being  used  to  provide  an 
alternate  solution  to  a  core  system  or  component  of  the  ACTD.  A  less  mature  alternate 
solution  may  be  available  that  offers  significant  benefits  (e.g.  higher  performance,  higher 
reliability  or  lower  cost).  The  Department’s  criteria  acknowledge  that  it  may  be  prudent 
to  support  parallel  evaluation  of  the  alternate  solution  if  the  outlook  is  favorable  and  the 
cost  is  modest.  Since  these  parallel  solutions  are  not  on  the  performance  critical  path, 
they  pose  no  risk  to  the  ACTD.  An  example  is  the  Outrider  heavy  fuel  engine  (HFE) 
which  was  also  discussed  on  page  six  of  the  draft  report.  The  HFE  was  an  attractive 
option,  but  it  was  not  a  requirement  for  this  ACTD. 
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The  GAO  draft  report  states:  “Another  example  of  the  inclusion  of  immature 
technology  occurred  in  the  Outrider  Unmanned  Aerial  Vehicle  project.  According  to 
program  documentation,  one  of  the  individual  technologies  to  be  incorporated  into  the 
vehicle  was  a  heavy  fuel  engine.  According  to  a  program  official,  it  was  later  deemed 
that  this  individual  technology  was  too  immature  and  an  alternate  technology  had  to  be 
used.” 

The  GAO  does  not  note  that  the  heavy  fuel  engine  (HFE)  was  an  alternate 
propulsion  option.  The  HFE  was  not  part  of  ACTD  requirement  established  by  the  Joint 
Requirements  Oversight  Council  (JROC).  Instead,  the  JROC  recommended  an 
aggressive  effort  to  develop  an  HFE  as  part  of  a  pre-planned  product  improvement  (P3I) 
program.  Similarly,  the  memorandum  signed  by  the  Under  Secretary  of  Defense 
(Acquisition  &  Technology)  did  not  require  an  HFE  in  the  basic  program  but  did  envision 
it  as  a  P3 1  effort.  The  company  that  won  the  ACTD  contract  proposed  to  use  a  gasoline 
engine  initially,  but  to  provide  an  HFE  replacement  if  it  performed  as  they  predicted. 
Early  in  the  program,  their  HFE  solution  was  determined  to  be  unworkable  and  all  efforts 
on  the  HFE  were  terminated.  While  the  HFE  was  an  attractive  option,  it  was  not  a 
requirement  for  this  ACTD.  It  was  simply  proposed  by  the  contractor  as  a  potentially 
beneficial  alternative,  if  it  worked.  The  baseline  engine  was  a  very  mature  gasoline 
engine  that  later  proved  to  have  insufficient  performance  to  offset  the  lower  than  expected 
lift  achieved  by  the  air  vehicle.  The  original  engine  was  subsequently  replaced  by 
another  mature,  but  higher  performing  engine.  All  engines  flown  in  the  program  were 
engines  that  had  previously  been  certified  and  were  already  in  production. 

The  Department  believes  the  core  technologies  in  both  the  CID  and  TUAV 
ACTDs  were  mature  and  had  been  properly  demonstrated  before  initiation  of  the 
programs. 

Reference  the  GAO  statement:  M... guidance  is  not  sufficient  to  ensure  that  on 
completion  of  the  basic  ACTD  demonstration,  a  military  capability  deemed  to  be 
worthwhile  would  enter  the  normal  acquisition  process  at  the  appropriate  point.” 
(p.5) 

DoD  Response:  The  Department  agrees  that  when  an  ACTD  is  determined  to  have 
sufficient  military  utility,  it  should  result  in  a  decision  to  field  the  residual  capability  and, 
if  additional  quantities  are  required,  the  program  should  enter  the  acquisition  process. 

The  objective  of  the  program  is  to  focus  on  priority  military  needs,  and  the  expectation  is 
that  if  the  ACTD  results  in  a  determination  of  high  military  utility,  the  capability  would 
be  fielded.  However,  in  the  case  of  some  ACTDs,  the  quantity  required  is  very  small  and 
the  residual  systems  may  fully  satisfy  the  need.  In  these  situations,  it  is  not  necessary  to 
enter  the  normal  acquisition  process.  In  addition,  the  decision  to  proceed  into  acquisition 
with  a  new  capability  is  based  on  many  factors,  among  which  are:  the  degree  of  military 
utility  the  proposed  capability  will  provide,  the  current  threat  projections,  and  the 
importance  of  other  capabilities  competing  for  resources.  It  would  therefore  not  be 
appropriate  for  OSD  to  specify  to  the  Services  which  specific  systems  they  should 
acquire.  However,  we  concur  it  is  appropriate  to  develop  guidance  which  requires 
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See  comment  6. 


See  comment  7. 


See  comment  8. 


See  comment  9. 


See  comment  10. 


ACTDs  to  demonstrate  high  military  utility,  and  where  additional  quantities  are  required 
to  satisfy  the  stated  need,  an  acquisition  decision  be  made  in  a  timely  manner. 

With  regard  to  the  appropriate  point  of  entry  into  the  acquisition  process,  the  DoD 
acquisition  process  encourages  tailoring  and  allows  a  program’s  entry  point  to  be  selected 
based  upon  the  circumstances  of  that  individual  program.  The  decision  criteria  for  entry 
of  an  ACTD  are  the  same  as  the  criteria  for  any  other  program  under  consideration.  It 
would  be  improper  to  establish  separate  criteria  for  systems  that  have  previously  been 
through  the  ACTD  process.  For  example,  to  enter  Low  Rate  Initial  Production  (LRIP), 
the  system  must  have  been  determined  by  the  operational  testers  to  be  potentially 
effective  and  suitable.  This  was  the  case  for  the  Predator  system.  The  term  “military 
utility”  is  defined  as  effectiveness  to  perform  its  mission,  suitability  to  be  issued  to  the 
operators,  and  overall  impact  on  the  military  operation.  The  GAO  report  is  incorrect  in 
stating  that  ACTDs  do  not  address  suitability.  Suitability  has  been  a  factor  since  the  start 
of  the  program  and  has  continued  to  grow  in  significance. 

The  Predator  ACTD  was  cited  by  the  GAO  as  a  program  that  did  not  enter  into  the 
correct  stage  of  development  after  completing  its  demonstration.  At  the  August  1997 
Defense  Acquisition  Board  Readiness  Meeting,  the  Defense  Acquisition  Executive 
approved  the  Predator  for  entry  into  the  formal  acquisition  process,  per  the 
August  8,  1997  Acquisition  Decision  Memorandum.  The  Predator  ACTD  was  the  first 
program  originating  as  an  ACTD  to  enter  the  formal  acquisition  process.  Following  the 
completion  of  that  ACTD,  a  working  group  was  established  with  representatives  from 
OSD  and  the  services  to  develop  guidelines  for  the  transition  of  ACTDs  into  the  formal 
acquisition  process.  The  objective  of  that  effort  was  to  apply  the  lessons  learned  during 
the  transition  of  Predator  to  subsequent  ACTDs.  The  resulting  changes  were  designed  to 
ensure  that,  when  the  objective  is  to  enter  into  LRIP,  the  necessary  preparations  are  made 
during  the  ACTD  to  ensure  the  system  design  and  testing  fully  support  that  objective. 
Many  parallels  with  commercial  practices  were  suggested  in  the  GAO  report,  but  the 
commercial  sector  does  not  design,  build  and  test  a  product  and  then  require  that  all  such 
products  be  redesigned  and  re-tested  prior  to  the  entry  into  production.  Instead,  they 
ensure  that  the  initial  design/build/test  phase  is  adequate  to  support  entry  into  production. 
Where  the  objective  is  to  transition  into  production,  this  same  approach  is  being  taken  in 
the  ACTD  program. 

The  alternative  recommended  by  the  GAO,  to  follow  an  ACTD  with  an  EMD 
program,  would  be  inefficient,  costly,  and  would  defeat  the  objective  of  reducing  the  time 
required  to  field  high  priority,  new  military  capabilities.  It  also  runs  counter  to  efforts 
underway  in  the  normal  acquisition  process  to  emphasize  a  single  development  phase 
prior  to  the  start  of  production.  Multi-phase  development  programs  for  military  systems 
can  take  two  to  four  times  longer  than  development  programs  in  the  commercial  sector. 
The  DoD  believes  the  proper  approach  is  to  design  the  development  process  to  result  in 
producible  systems  in  a  single  development  phase.  Since  ACTDs  begin  the  process  with 
mature  technologies  and  include  not  only  development  testing,  but  extensive  operational 
testing  as  well,  pursuing  a  single  development  phase  is  a  prudent  approach. 
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See  comment  11. 


See  comment  12. 


Additionally,  it  should  be  noted  that  the  Predator  system  has  performed  extremely 
valuable  work  in  the  Department’s  current  peace  enforcement  operations  in  Bosnia*  The 
system  has  been  praised  highly  by  CINCEUR  and  others. 

Reference  the  GAO  statement:  “DoD’s  current  practice  of  procuring  prototypes 
beyond  those  needed  for  the  basic  ACTD  demonstration  and  before  completing 
product  and  concept  development  and  testing  is  unnecessarily  risky.”  (p.6) 

DoD  Response:  For  46  of  the  48  ACTDs  initiated  to  date,  the  procurement  of  prototypes 
has  been  limited  to  those  needed  for  the  basic  ACTD  demonstration.  However,  for  the 
two  remaining  ACTDs,  following  that  approach  would  preclude  fielding  a  residual 
capability  following  the  ACTD.  In  both  cases,  an  Army  company  will  be  fully  trained 
and  equipped  with  a  new  missile  system  for  evaluation  during  the  ACTD.  However, 
since  all  of  the  initially  acquired  missiles  would  be  expended  during  the  ACTD,  there  will 
be  none  left  to  provide  a  limited  operational  capability  at  the  conclusion  of  the  ACTD. 
Ironically,  the  cost  of  the  additional  missiles  is  a  very  small  fraction  of  the  cost  of  the 
ACTD,  but  without  missiles  there  is  no  residual  capability.  The  soldiers  would  have  to 
be  reassigned  and  the  fire  units  and  all  supporting  equipment  placed  in  storage.  An 
opportunity  to  reduce  the  time  to  field  an  important  new  capability  would  be  lost.  To 
prevent  this  situation,  these  programs  were  structured  to  include  a  procurement  of  a 
second  lot  of  missiles,  consisting  of  the  minimum  number  necessary  to  equip  a  company. 
The  second  procurements  were  scheduled  to  occur  after  the  initial  lot  of  missiles  had 
completed  sufficient  testing  to  confirm  readiness  for  limited  production.  The  first 
instance  was  the  Enhanced  Fiber  Optic  Guided  (EFOG)  Missile.  Due  to  delays  in  the 
development  and  testing  of  that  missile,  there  was  insufficient  test  data  to  support  a 
decision  to  initiate  follow-on  production.  For  that  reason,  the  Army  decided  not  to 

in 

FY02  for  the  Line-of-Sight  Anti-Tank  (LOS  AT)  missile  program.  The  LOS  AT  ACTD 
was  restructured  to  ensure  that,  prior  to  initiation  of  follow-on  procurement,  all  required 
component,  subsystem,  and  system  flight-testing  would  be  completed.  The  utility 
assessment,  which  includes  the  supportability  assessment  and  the  validation  of 
requirements,  will  be  completed  prior  to  the  2-year  post  demonstration  period  when  the 
initial  procurement  occurs. 

The  criteria  for  the  limited  procurement  decisions  were  explicitly  defined, 
reviewed  and  accepted  by  the  appropriate  officials  within  the  Army  who  are  responsible 
for  such  procurement  decisions.  The  LOSAT  system  is  designed  to  enhance  the  early 
entry  forces  by  providing  decisive  firepower  to  counter  armored  opposition.  Limited 
quantities  are  envisioned.  If  any  follow-on  procurement  is  required  beyond  the  initial 
complement  of  missiles  discussed  above,  that  decision  can  be  made  prior  to  completion 
of  the  initial  procurement. 


proceed  with  follow-on  procurement  and  removed  the  procurement  funds  from  the 
planned  year.  The  Army  subsequently  determined  that  a  similar  situation  would  o\ 
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The  phasing  for  the  follow-on  procurement  of  LOSAT  missiles  has  been  carefully 
structured  to  reduce  the  time  required  to  field  this  capability,  but  to  ensure  that  all 
necessary  information  is  available  to  support  the  procurement  decision.  Further  delays 
would  result  in  no  additional  data  being  collected  and  no  reduction  in  risk,  but  would 
preclude  having  an  early  operational  capability.  It  would  also  result  in  the  company  of 
trained  LOSAT  soldiers  being  dis-established  and  re-established  two  years  later. 

Recommendations 

GAO  Recommendation  1:  That  the  Secretary  of  Defense  specify  in  the  program 
guidance  a  tangible  metric  to  define  the  maturity  of  the  technology  to  be  used  in 
each  Advanced  Concept  Technology  Demonstration  project. 

DoD  Response:  Partially  concur.  A  tangible  metric  to  define  the  maturity  of  technology 
is  appropriate.  However,  in  response  to  a  similar  recommendation  by  the  DoD  IG  in 

1 997,  a  metric  for  technological  maturity  was  established.  The  metric  was  published  in 
the  Defense  Acquisition  Deskbook  in  April  1998  and  in  the  ACTD  Master  Plan  in  June 

1998.  This  metric  has  satisfactorily  addressed  the  DoD  IG’s  concerns,  and  has  been 
applied  to  all  ACTDs  in  FY97-99.  This  criterion  also  addresses  technology  maturity  for 

See  comment  1  the  reclu*red  performance  levels  and  operating  environments,  which  are  critical.  The 

report  cites  no  examples  of  immature  technology  in  ACTDs  initiated  since  1996. 

Recommendation  2:  That  the  Secretary  of  Defense  clearly  state  in  the  program 
guidance  that,  if  the  demonstrated  concept  or  equipment  has  military  utility,  the 
Service(s)  involved  should  tailor  an  appropriate  program  to  complete  product  and 
concept  development  and  testing  before  production  begins. 

DoD  Response:  Partially  concur.  It  is  appropriate  for  the  Secretary  of  Defense  to  state 
in  program  guidance  that,  where  additional  quantities  are  required,  an  acquisition 
decision  will  be  made  in  a  timely  manner.  However,  the  decision  for  a  specific  program 
to  proceed  into  acquisition  should  be  based  on  many  factors,  among  which  are:  the 
degree  of  military  utility  the  proposed  capability  will  provide,  the  current  threat 
projections,  and  the  importance  of  other  capabilities  competing  for  resources.  Directing 
the  Services  to  proceed  into  acquisition  based  solely  on  military  utility  would  not  be 
appropriate.  In  addition,  the  entry  point  into  the  acquisition  process  should  be  selected 
based  upon  the  circumstances  of  each  individual  program,  using  the  same  criteria  that  is 
used  to  determine  the  entry  point  of  any  program  entering  that  process.  Requiring  that  all 
ACTDs  must  undergo  further  product  and  concept  development  and  testing  prior  to  a 
production  phase  would  seriously  undermine  one  of  the  key  objectives  of  the  ACTD 
program  -  reducing  acquisition  cycle  time.  The  Department  feels  the  proper  approach  is 
to  ensure  that  ail  ACTDs  contain  all  product  and  concept  development  and  testing 
See  comment  8.  necessary  to  support  the  planned  entry  point  into  the  acquisition  process.  This  approach 

is  reflected  in  the  published  guidelines  for  transition  of  ACTDs  into  acquisition.  The 
DoD  recommends  that  the  GAO  consider  this  approach  and  the  significant  benefits  from 
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See  comment  12. 


both  cost  and  schedule  standpoints,  as  well  as  commercial  sector  evidence  supporting  this 
approach. 

Recommendation  3:  The  Secretary  of  Defense  limit  the  number  of  prototypes  to  be 
procured  to  the  quantities  needed  for  early  user  demonstrations  of  mature 
technology. 

DoD  Response:  Partially  concur.  The  number  of  prototypes  to  be  procured  in  an  ACTD 
should  normally  be  limited  to  the  number  needed  for  the  assessment  of  military  utility. 
Any  additional  procurements  required  to  field  a  limited  operational  capability  should  be 
authorized  by  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  after 
confirming  that  sufficient  testing  has  been  satisfactorily  completed  to  support  the  limited 
procurement. 
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The  following  are  gao’s  comments  on  the  Department  of  Defense’s  (dod) 
letter,  dated  August  31,  1998. 


GAO  Comments 


1.  dod’s  guidance  is  improved  over  previous  versions,  dod’s  revised 
guidance  on  the  maturity  of  technology  to  be  used  in  the  Advanced 
Concept  Technology  Demonstration  (actd)  states  that 

“. . .  new  technologies  proposed  for  incorporation  into  an  actd  should  not  be  in  the  6.1 
(basic  research)  or  6.2  (applied  research)  budget  categories.  Furthermore,  the  technologies 
must  have  been  successfully  demonstrated  at  the  subsystem  or  component  level  and  at  the 
required  performance  level  prior  to  the  start  of  the  actd.” 


The  new  guidance,  however,  permits  the  selection  of  immature 
technology — even  as  the  primary  or  core  technology — provided  that  it  is 
demonstrated  prior  to  the  actd  demonstration.  The  guidance  goes  on  to 
describe  several  types  of  exceptions  under  which  immature  technologies 
may  be  permitted  to  be  used  in  an  actd.  Moreover,  the  revised  guidance 
would  permit — as  has  occurred  in  some  cases — the  approval  of  actd 
projects  without  the  technologies  having  been  identified.  Our  point  is  that 
dod  should  continue  to  focus  the  actd  program  on  the  use  of  mature 
technology  with  few,  if  any,  exceptions. 

2.  For  the  Combat  Identification  actd,  the  management  plan  for  the  project 
shows  that  the  technologies  in  question  were  integral  parts  of  the 
capabilities  being  evaluated.  The  demonstration  manager  told  us  that  the 
involvement  of  immature  technologies  was  one  of  the  reasons  that  the 
project  was  behind  schedule. 

3.  According  to  the  Outrider  project’s  management  plan,  the  heavy  fuel 
engine  was  the  primary  technology  to  be  used,  not  the  alternative.  Also, 
the  involvement  of  this  immature  technology  was  one  of  the  causes  of 
schedule  slippage  and  cost  overruns  on  the  project. 

4.  dod’s  most  current  actd  guidance  states 

“. .  .Strategies  and  approaches  are  described  to  facilitate  transitioning  from  an  actd  to  the 
acquisition  process  as  defined  in  dod  5000.2R.  The  suggested  approaches  are  based  on 
lessons  learned.  The  focus  of  the  suggestions  are  actds  that  are  planned — if  successful — to 
enter  the  acquisition  process  at  the  start  of  LRIP.” 
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Although  there  is  a  basic  recognition  that  the  transition  to  development 
may  be  possible,  the  bulk  of  the  guidance  is  on  how  and  when  to  transition 
to  production.  As  pointed  out  in  the  report,  the  guidance  does  not  describe 
when  a  transition  to  development  or  what  types  of  development  activity 
may  be  appropriate.  In  our  view,  the  guidance  needs  to  be  more  balanced 
between  the  possibility  of  transition  to  development  and  the  transition  of 
actd  projects  directly  to  production. 

5.  As  discussed  in  the  report,  the  independent  operational  testing  agencies 
are  observers  in  the  actd  demonstrations  and  not  active  participants. 

While  the  Office  of  the  Director  of  Operational  Test  and  Evaluation  was  an 
observer  during  the  Predator  demonstration,  a  determination  was  not 
made  that  Predator  was  potentially  effective  and  suitable. 

6.  We  agree  that  actds  address  the  technology’s  suitability.  However,  the 
actd  focus  on  suitability  is  in  a  very  general  sense  and  extensive  data  is 
not  collected  on  the  system’s  reliability,  maintainability,  and  other  aspects 
of  suitability  needed  to  support  production  decisions. 

7.  As  our  report  states,  the  Predator  was  rushed  into  low-rate  initial 
production  prematurely  given  the  limited  amount  of  testing  conducted  at 
that  time  and  the  problems  that  were  uncovered  during  that  limited 
testing. 

8.  dod’s  equating  a  determination  of  military  utility  (based  on  an  actd 
demonstration)  with  a  determination  of  a  system’s  readiness  to  begin 
production  is  inappropriate  because  production  decisions  require  more 
testing  data.  During  our  review,  we  noted  that  sufficient  information  was 
not  obtained  from  an  actd  demonstration  to  make  a  commitment  to 
limited  production.  Commercial  practice  would  dictate  that  much  more 
information  be  obtained  about  a  product’s  effectiveness,  suitability, 
producibility,  or  supportability  before  such  a  commitment  is  made.1  We 
believe  the  actd  guidance  needs  to  be  more  balanced  and  should 
anticipate  that  actd  prototypes  may  need  to  conduct  more  product  and 
concept  development  and  testing  prior  to  production.  We  have  long  held 
the  view  and  have  consistently  recommended  that  dod  use  extreme 
caution  to  avoid  premature  commitments  to  production. 

9.  We  are  not  suggesting  that  a  lengthy  development  phase  be  conducted 
on  all  actd  products  nor,  as  dod  appears  to  suggest,  that  an  actd  prototype 


JBest  Practices:  Successful  Application  to  Weapon  Acquisitions  Requires  Changes  in  DOD’s 
Environment  (GAO/NSIAD-98-56,  Feb.  24,  1998). 
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may  be  ready  to  start  limited  production  immediately  after  its  basic 
demonstration.  As  dod  stated  in  its  intent  to  establish  the  actd  program, 
we  believe  the  benefit  of  the  actd  process  is  in  eliminating  or  reducing 
technology  development,  not  in  making  early  commitments  to  production 
or  in  postponing  product  and  concept  development  and  testing  activities 
until  after  production  starts. 

10.  While  actd  demonstrations  are  performed  in  operational  environments, 
they  are  not  operational  tests.  During  the  course  of  our  work,  we  held 
several  discussions  with  officials  from  the  operational  test  community. 
Those  officials  were  in  favor  of  the  user  demonstrations  featured  in  the 
actd  program,  but  none  considered  those  demonstrations  as  substitutes 
for  operational  testing  because  of  their  informality,  lack  of  structure,  and 
the  lack  of  a  defined  requirement  by  which  to  measure  performance. 

11.  dod  appears  not  to  recognize  the  very  real  possibility  that  the  actd 
demonstration  may  find  the  technology  in  question  to  have  little  or  no 
military  utility  or  to  be  unaffordable  in  today’s  budgetary  and  security 
environment.  In  fact,  due  to  budget  constraints,  the  Army  was  forced  to 
prioritize  its  procurement  programs,  and  the  planned  procurement  funding 
for  Enhanced  Fiber  Optic  Guided  missiles  has  been  reallocated. 

12.  While  we  agree  with  dod  that  test  results  should  form  the  basis  for 
starting  limited  procurement,  the  testing  needed  goes  beyond  the  basic 
demonstration  of  military  utility  provided  by  the  actd  program. 


(707299) 
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